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This paper presents a comprehensive 
method for evaluating intrusion detection 
systems (IDSs). It integrates and extends 
ROC (receiver operating characteristic) 
and cost analysis methods to provide an 
expected cost metric. Results are given for 
determining the optimal operation of an 
IDS based on this expected cost metric. 
Results are given for the operation of a 
single IDS and for a combination of two 
IDSs. The method is illustrated for: 1) 
determining the best operating point for a 
single and double IDS based on the costs 
of mistakes and the hostility of the operat- 
ing environment as represented in the prior 
probability of intrusion and 2) evaluating 
single and double IDSs on the basis of 



expected cost. A method is also described 
for representing a compound IDS as an 
equivalent single IDS. Results are present- 
ed from the point of view of a system 
administrator, but they apply equally to 
designers of IDSs. 
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1. Introduction 

Little was done to evaluate computer intrusion detec- 
tion systems (IDSs) prior to the evaluations conducted 
by the Massachusetts Institute of Technology's Lincoln 
Laboratory under the sponsorship of the DARPA in 
1998. This effort is known as the 1998 DARPA off-line 
intrusion detection evaluation. It was the first compre- 
hensive test of multiple IDSs using a realistic setting. 
Various accounts of this evaluation have been pub- 
lished by Durst et al. [1], McHugh [2], Lippmann et al. 
[3], Stolfo et al. [4], and McHugh et al. [5]. This evalu- 
ation was the first that evaluated many IDSs, used a 
wide variety of intrusions, simulated realistic normal 
activity, and produced results that could be shared by 
many researchers. 

During the 1998 DARPA evaluation, detection 
results were combined with the total number of net- 



work sessions to give two summary measures of an 
IDS's performance: detection rate (intrusions detected 
divided by intrusions attempted) and false alarm rate 
(false alarms divided by total network sessions). These 
summary measures were taken as an estimate of one 
point on the IDS's receiver operating characteristic 
(ROC) curve. A ROC curve is a plot of detection prob- 
ability versus false alarm probability. It shows the prob- 
ability of detection provided by the IDS at a given false 
alarm probability. Alternatively, it shows the false 
alarm probability provided by the IDS at a given prob- 
ability of detection. 

Lippmann et al. [3] claim, "a novel feature of this 
evaluation is the use of receiver operating characteris- 
tic (ROC) techniques to evaluate intrusion detection 
systems." Although Lippmann et al. [3] used ROC 
curves, their evaluations were based on simply compar- 
ing ROC curves for dominance. A dominant curve 
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would lie above and to the left of a dominated curve. 
No metric was presented for the degree of dominance, 
nor was any statement made as to the value of one IDS 
over another or the value of an IDS over no IDS. 
Others, however, have proposed metrics for evaluating 
the ROC curves of IDSs. Durst et al. [1] contend that, 
"the area under the curve is one measure of an intrusion 
detection system's effectiveness." Axelsson [6] propos- 
es a "required level of false alarms;" Durst et al. [1] 
suggest a false alarm rate that is "manageable." 
Saydjari [7] proposes a goal on detection probability 
and probability of false alarm. Presumably, metrics 
could be developed (e.g., Euclidean distance) that 
describe how "close" a given ROC curve is to the 
required level or goal. However none of these metrics 
is satisfactory in that none provides a complete meas- 
ure of the capability of an IDS. 

Stolfo et al. [4] propose an alternative method for 
evaluating IDSs that is based on cost metrics. They 
claim to, "demonstrate that the traditional statistical 
metrics used to train and evaluate the performance of 
learning systems (i.e., statistical accuracy or ROC 
analysis) are misleading and perhaps inappropriate for 
this application." They claim that their cost-based met- 
rics are more appropriate, and they fiirther, "demon- 
strate how the [cost-based] techniques developed for 
fraud detection can be generalized and applied to the 
important area of intrusion detection." They apply their 
cost-based methods by calculating the total costs 
incurred with different IDSs by adding the costs from a 
number of simulation trials. They do not show how 
their method uses all of the information in a ROC 
curve, nor do they provide a compelling demonstration 
of the superiority of the cost metric. 

We demonstrate that both the ROC analysis and 
other cost analysis methods that we have reviewed are 
incomplete. Furthermore, we demonstrate how a deci- 
sion tree can combine and extend the ROC and cost 
analysis methods to provide an expected cost metric 
that reflects the intrusion detection system's ROC 
curve, cost metrics, and an assessment of the hostility 
of the environment as summarized in the prior proba- 
bility of intrusion. We further demonstrate how this 
method can be used to: decide the optimal operating 
point on an IDS's ROC curve, choose the best intrusion 
detection system, determine the value of one intrusion 
detection system over another, determine the value of 
an IDS over no IDS, and determine how to adjust the 
operation of an IDS to respond to changes in its envi- 
ronment. 

McHugh's [2] very thorough critique of the 1998 
DARPA evaluation raises a number of serious ques- 
tions about how the ROC curves in it were constructed. 



He also raises concerns about the appropriateness of 
ROC analyses for these evaluations at all, especially if 
the unit of measurement is different for different IDSs. 
We do not address how the ROC curves are obtained; 
we show how they should be compared once they have 
been obtained. 

This paper is arranged as follows. Section 2 
describes our method for evaluating a single IDS. It 
describes ROC curves, presents a decision tree analysis 
for determining an IDS's optimal operating point, and 
shows how the expected cost of operating an IDS in a 
hostile environment can be used to evaluate an IDS. 
Section 2 also describes a method for determining the 
expected value of one IDS over another. We demon- 
strate that this expected value depends on the costs of 
mistakes, the probability of intrusion, and the IDSs' 
ROC curves, not just some of these factors. We demon- 
strate that the area under a ROC curve is not a valid 
measure of an IDS's effectiveness, contrary to the 
assertions of Durst et al. [1]. 

Section 3 extends the method to evaluate a com- 
pound IDS that consists of two independent IDSs. 
Results are presented that describe the optimal opera- 
tion of the combination of two IDSs and compare the 
expected cost from a single IDS with that from a com- 
pound IDS. Results are shown for a compound IDS 
composed of two independent identical IDSs, two inde- 
pendent different IDSs, and two independent IDSs, one 
with a zero probability of false alarms. 

Section 4 describes how a compound IDS can be rep- 
resented by a single, composite ROC curve that is 
derived from the ROC curves of its components. 

Section 5 presents conclusions, recommendations, 
and suggested extensions of the method. 

Four appendices contain technical details. Appendix 
A (Sec. 6) shows the analysis for a compound IDS with 
a single decision. Appendix B (Sec. 7) shows the analy- 
sis for a compound IDS with sequential decisions. 
These appendices show that the expected cost from 
using a compound IDS composed of two independent 
IDSs is the same regardless of whether the response 
decision is made sequentially after each component 
IDS's report or if the response decision is made only 
once on the basis of both reports. Appendix C (Sec. 8) 
shows simplified analyses and the geometry of the 
ROC. It describes the conditions under which the 
embedded decision can be removed from the decision 
tree, describes an analysis of the ROC convex hull, and 
describes an extension of the analysis that includes 
additional costs. Appendix D (Sec. 9) shows the deriva- 
tion of a single, composite ROC curve to represent the 
performance of multiple IDSs. 
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2. Evaluation of a Single Intrusion 
Detection System (IDS) 

A computer intrusion detection system (IDS) is con- 
cerned with recognizing whether an intrusion is being 
attempted into a computer system. An IDS provides 
some type of alarm to indicate its assertion that an 
intrusion is present. The alarm may be correct or incor- 
rect. A decision maker (e.g., the system administrator) 
can decide to respond to the alarm or to ignore the 
alarm. This section describes a decision analysis 
method for determining the best operating point for an 
IDS and an expected cost metric that can be used to 
evaluate an IDS. 

An IDS's receiver operating characteristic (ROC) 
curve describes the relationship between the two oper- 
ating parameters of the IDS, its probability of detection, 
1-p, and its false alarm probability, a. That is, the ROC 
curve displays the l-p provided by the IDS at a given 
a. It also displays the a provided by the IDS at a given 
l-j3. The ROC curve thus summarizes the performance 
of the IDS. We do not address how one generates this 
ROC curve, just what to do with it after it is deter- 
mined. 

Figure 1 shows two possible ROC curves that are 
used in this paper These are similar to two ROC curves 
that were determined by Graf et al. [8] from actual data 
in the 1998 DARPA off-line intrusion detection evalua- 
tion. IDS E's ROC curve is similar to the ROC curve 
for the EMERALD (Event Monitoring Enabling 
Responses to Anomalous Live Disturbances [9]), and 
IDS C's ROC curve is similar to the ROC curve for the 
Columbia IDS [10]. IDS "C" is shown with five dis- 
crete operating points, and IDS "E" is shown with four 
The lines shown connecting the points are added as a 
visual aid to the reader but are irrelevant to describing 
the performance of the IDSs. Gaflfney and Ulvila [11] 
show that one would never choose to operate an IDS at 
an interior point on the line segment connecting two 
operating points. 

The following nomenclature is used throughout this 
paper The system can be in one of two states or condi- 
tions: either with an intrusion present (I) or with no 
intrusion present (NT). The prior probability of an intru- 
sion is called p. The IDS reports either an intrusion 
alarm (A) or no alarm (NA). The parameters of the 
IDS's ROC curve are: the probability of an alarm given 
an intrusion, the detection probability, P{A\T) = I- p 
(or the probability of no alarm given an intrusion, 
/'(NA|I) = p), and the probability of an alarm given no 
intrusion, the false alarm probability, /'(A|NI) = a. 
Thus, a and p are the probabilities of the two types of 
reporting errors. 
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Fig. 1. ROC curves. 



Either report from the IDS will trigger one of two 
actions: either respond as though there were an intru- 
sion (R) or do not respond (NR). Consequences of the 
combinations of possible actions and states of the sys- 
tem are specified by the costs of errors. The cost of 
responding as though there were an intrusion when 
there is none is denoted C„. The cost of failing to 
respond to an intrusion is denoted Cp. Without loss of 
generality, we can rescale costs by defining a cost ratio, 
C= Cp IC„. The analyses in the body of this paper 
assume that the costs of correct responses are zero. 
Section 8.3 describes how these analyses could be 
extended to the general situation with costs for all com- 
binations of actions and states of the system. 

In practice, these costs are estimated by considering 
the consequences of the errors, and costs will be differ- 
ent for different computer systems and for different 
operating conditions. For example, C„ includes the 
obvious cost of the person who responds to the alarm 
and the not-as-obvious cost to the users due to the 
degraded performance of the computer system while 
the alarm is being investigated. These costs depend on 
the nature of the response. Common responses include: 
filtering, isolation, changing logging or other proce- 
dures, or disconnection [1], and some of the responses 
could be automated. Cp is the cost of the damage done 
by the intruder while he remains undetected. It includes 
the cost to restore the computer system to its undam- 
aged condition. For critical systems, it could include 
the costs of errors committed by the system while under 
the influence of the intruder (e.g., launching a missile 
or shutting down a power grid). In the analysis present- 
ed here, point-estimates are used for costs. An exten- 
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sion could use probability distributions over the costs, 
but the results, which are based on expected costs, 
would be similar 

In general, companies are reluctant to share informa- 
tion about their costs, but a procedure such as the fol- 
lowing could be used by an organization to estimate 
these costs. The cost of various actions, such as 
responding to an alarm might be estimated by a careful 
consideration of the steps that would be taken to 
respond to one. The cost of ignoring an alarm when 
there actually is an intrusion into the system might be 
estimated in part by an analysis of the data available 
from surveys such as the 2002 CSI/FBI Computer 
Crime and Security Survey [12], and in part by a care- 
ful analysis of the cost or impact on the system and 
organization protected by the IDS. Industry data, such 
as those available from a survey, can suggest a value or 
range of values. However, such "industry data" cannot 
be a completely satisfactory substitute for a careful 
analysis of one's own organization or business. This sit- 
uation is analogous to the estimation of software devel- 
opment costs. One might use "canned" data, such as 
available from a commercial tool or what one obtains 
from discussions with other organizations' personnel or 
from published papers or books. However, it is always 
preferable to use data from one's own organizational 
experience as the basis of an estimate. 

The expected cost of any operating point of the IDS 
is determined by analyzing the decision tree shown in 
Fig. 2. This decision tree shows the sequence of actions 
(squares) and uncertain events (circles) that describe 
the operation of the IDS and of the actions or respons- 
es that can be taken, based on reports. It also shows the 
consequences of the combinations of actions and 
events. The costs shown correspond to the conse- 
quences. The convention in a decision tree is to read it 
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Fig. 2. Decision tree of the IDS's expected cost. 



from left to right. The path leading to any point in the 
tree is shown to the left of the point and is assumed to 
be determined. Paths to the right of any point show all 
subsequent possibilities, which are not yet determined. 

This decision tree shows that the optimal decision 
may be to take the action opposite of the one recom- 
mended by the IDS. That is, it may be optimal to ignore 
an alarm or to respond to a case of no alarm. Section 8. 1 
describes the conditions under which the optimal deci- 
sion is to follow the IDS's recommendation. 

Decision or action nodes, which are displayed as 
squares, are under the control of the decision maker 
The decision maker will choose which branch to fol- 
low. Event nodes, which are shown as circles, are not 
under the control of the decision maker but are subject 
to uncertainty. A probability distribution represents the 
uncertainty about which branch will happen following 
an event node. Associated with each uncertain event is 
its probability of occurrence. There are three probabili- 
ties specified in the tree: 

Pi = the probability that the IDS reports an alarm, 
P2 = the conditional probability of intrusion given that 

the IDS reports an alarm, and 
/?3 = the conditional probability of intrusion given that 

the IDS reports no alarm. 
Gaffney and Ulvila [11] show how these probabilities 
can be derived from the values of a, (3, and p. 

The expected cost of an operating point is calculated 
by "rolling back" the decision tree [13] shown in Fig. 2. 
Working from right to left, the expected value at an 
event node is calculated as the sum of products of prob- 
abilities and costs for each branch. The expected cost at 
an action node is the minimum of expected costs on its 
branches. 

An operating point for an IDS is defined as the val- 
ues of the parameters a and p. Gaffney and Ulvila [11] 
show that the expected cost of operating at a point on 
an IDS's ROC curve is: Mm{CI3p, (1 - a)(l -p)} + 
Min{C(l -li)p, a(\ -p)}, where C= Cfj/C„ and p is 
the prior probability of intrusion. 

Choosing the best operating point is important 
because IDSs can often be adjusted to operate at differ- 
ent points. Lippmann et al. [3] state: "most intrusion 
detection systems provide some degree of configura- 
tion to allow experts to customize the system to a given 
environment." Axelsson [6] notes: "the performance 
point of the IDS can be tuned to meet the requirements 
of the operating environment." Kent [14] states: "many 
systems have the equivalent of a tuning knob that 
allows a system administrator to adjust the sensitivity 
of the [intrusion detection system]." 
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The decision of choosing an operating point is to 
select the point with the least expected cost. That is, the 
values of a and ^8 are chosen to minimize expected 
cost. The problem is to choose a and ^ on the ROC 
curve so as to minimize (for given values of C and/)): 
Mm{C^p, (1 - a)(l -p)} + Min{C(l - ^)p, a(l -/?)}. 

Figure 3 shows, for IDS "C", the relationship 
between the optimal operating point and the environ- 
ment in which the IDS is to operate and the expected 
cost of operating at that point. It also shows the optimal 
response to an alarm. Figure 3 was determined for a 
cost ratio of 500. That is, if it is 500 times as expensive 
to fail to respond to an intrusion as it is to respond to a 
false alarm. Labels beneath the horizontal axis in Fig. 3 
indicate that if the prior probability that a given attempt 
to use the system is an intrusion is less than 6.7 x 10"*, 
then it is best to never respond to an alarm. However, if 
the prior probability of an intrusion is greater than 
7.1 X lO"'', then it is best to treat every attempt to use 
the system as though it were an intrusion. In between, 
it is best to respond to an alarm from the IDS. 

The solid lines in Fig. 3 show the ranges over which 
the optimal operating point is the one shown on the 
right vertical axis. For example, if the prior probability 
of an intrusion is between 6.7 X 10~* and 1.0 x 10"*, 
then the optimal operating point is a=2x 10"^ and 
1 - j8 = 0.60. Continuing, if the prior probability of an 
intrusion is between 1.0 X 10"* and 2.0 x 10"*, then the 
optimal operating point is at a= 5 X 10"^ and I - P = 
0.66, and so forth. 

The curve in Fig. 3 shows the expected cost (along 
the left vertical axis), in units of the cost of a false 
alarm, for each attempt to use the system when the IDS 
is operating at the optimal point and the optimal 



response decision is taken. The cost rises from 5.0 x 
10"* when the prior probability of intrusion is 1.0 X 10"* 
to 1.4 X 10"^ when the prior probability of intrusion is 
1.0 X 10^ to 0.99 when the prior probability of intru- 
sion is 1.0 X 10"^ (scales are logarithmic in Fig. 3). 

The environment at the 1998 DARPA Off-line 
Intrusion Detection Evaluation was meant to simulate 
realistic normal traffic on a computer network at an Air 
Force base [1]. In this environment, there were 43 
intrusion attempts out of 660 000 network sessions in a 
one-day period. This translates to a base-rate of intru- 
sion of 43/660 000 = 6.52 X 10"^ per session. If the IDS 
is applied each session and intrusion responses are on a 
per-session basis, then, if we estimate the prior proba- 
bility of intrusion as the base-rate, /? = 6.52 x 10"^. 
Figure 3 shows that, at this prior probability of intru- 
sion, the best decision is to respond to an alarm from 
the IDS, the expected cost is 0.009, and the best setting 
for the IDS is at a = 15 X 10"^ and 1 - ^8 = 0.72. 

The expected costs of different IDSs can be com- 
pared by subtracting the expected costs for the IDSs 
when each is operating at its optimal point. For any 
given cost ratio, C, and prior probability of intrusion, /», 
the optimal operating point will be different for IDSs 
with different ROC curves. Furthermore, the expected 
costs will differ for different ROC curves. The differ- 
ence in expected cost provides an expected value met- 
ric for comparing the two IDSs. 

In practice, one might be faced with the choice from 
among several different IDSs that offer different per- 
formances that can be characterized by different ROC 
curves. The analysis presented here provides a way to 
determine which ROC curve, and thus which IDS, is 
best. It also quantifies the preference in terms of a 




Fig. 3. Optimal operating points and expected cost for IDS "C (when cost ratio is 500). 
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difference in expected cost. The choice of a preferred 
ROC curve and the degree of that preference depend on 
the operating environment as characterized hy p and C. 
Consider the two ROC curves shown in Fig. 1 . Since 
the ROC curve for IDS "C" lies above and to the left of 
the ROC curve for IDS "£", and since these curves do 
not intersect, IDS "C" is always better than IDS "£". 
However, the value of that improvement, which is due 
to a smaller expected cost, depends on the values of C 
and p. Figure 4 summarizes the result. If C= 500 (i.e., 
if the cost of failing to respond to an intrusion is 500 
times the cost of responding to a false alarm), then IDS 
"C" is preferred over IDS "£" for values of/? less than 
0.0071. The maximum difference in expected cost is 
0.42 when/. = .0042. If C = 1000, then IDS "C" is pre- 
ferred for values of p less than 0.0036, and the maxi- 
mum difference in expected cost is 0.42 when p = 
0.0021. 
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Fig. 4. Expected value of IDS "C over IDS "i?" for different values 
of C and/). 



3. Evaluation of Multiple Intrusion 
Detection Systems (IDSs) 

The same type of analysis can be used to evaluate 
multiple IDSs operating in series or in parallel to eval- 
uate the traffic on a system. In the case of two IDSs 
operating in a manner such that the results from both 
IDSs are known before the decision of whether to 
respond is made, the decision tree is as shown in Fig. 5. 
(Appendix B, Sec. 7, shows that the results are the same 
regardless of whether a single response decision is 
made on the basis of both IDSs' reports or if response 
decisions are made sequentially after the receipt of each 
IDS's report.) This decision tree is read the same way 
as the decision tree for a single IDS. The first uncertain- 
ty is the report from each IDS, an alarm or no alarm 
from IDS 1 (Al or NAl) and IDS 2 (A2 or NA2). Next 
is the decision to respond or not. The next uncertainty 



is whether the actual condition is either an intrusion or 
no intrusion. Costs are the costs of errors — either 
responding to a false alarm (CJ or failing to respond to 
an intrusion (Cp). The cost ratio, C= Cp /C„. The 
parameters of this analysis are the probabilities of the 
reports, Pi, P2, Pi, and p^ and the probabilities of intru- 
sion conditional on the reports, q^, q2, q^, and q^. 
Section 6 shows that, if the two IDSs are independent, 
then the expected cost for the two-IDS decision tree, in 
terms of the parameters of the two ROCs («!, a^. Pi, 
and P2), the prior probability of intrusion (p), and the 
cost ratio (Q is: 

Min{(l -p)aia2,Cp(l - A)(l - Pi)} + Min{(l - 
/?)«!(! - a2),Cpi\ - PdPi} + Min{(l -p)i\ - a^a^, 
CM(1 - A)} +Min{(l -p)i\ - «,)(! - «2), CMA}- 

3.1 Two Identical IDSs 

The results of the analysis for two IDSs can be dis- 
played in a fashion similar to the results for a single 
IDS. Figure 6 shows the results for two IDSs with iden- 
tical, independent ROCs, when each IDS has the per- 
formance of IDS "C" and the cost ratio is 500. This 
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Fig. 5. Decision tree for a compound IDS consisting of two IDSs. 
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could be the case if two IDSs used completely different 
methods of detection yet provided identical perform- 
ance as evidenced by identical ROC curves. The stream 
of incoming traffic could be examined separately by 
each IDS, and each IDS would provide a separate 
alarm. Figure 6 shows the relationship between the 
optimal operating point and the environment in which 
the IDS is to operate and the expected cost of operating 
at that point. It has some interesting properties when 
compared with the analogous Fig. 3 for a single IDS. 
First, the "double IDS" is better than none over a larg- 
er range on the prior probability of intrusion, p, from 
1.0 X 10"" to 0.025. \fp is below the lower limit, it is 
best to never respond to an alarm. If j? is higher than the 
upper limit, it is best to respond to every attempt to use 
the system as though it were an intrusion. In between 
these limits, it is best to respond only if both IDSs indi- 
cate an alarm for values of p up to 2.5 x 10"^, and to 
respond to an alarm from either IDS above this value of 

P- 

In the case of two IDSs, each IDS can be set inde- 
pendently so that the combined performance is optimal. 
This results in two different settings, one for each IDS. 
As the prior probability of intrusion increases, the opti- 
mal settings of the false alarm probabilities of the two 
IDSs («! and a^ increase as shown by the right-hand 
axis in Fig. 6. (See Fig. 1 for the value of 1 - j3 at each 



value of a.) Increases are usually changes in a single 
IDS's setting, but sometimes the settings for both IDSs 
change. Once the value ofp increases above 2.5 x 10"', 
the optimal false alarm rates revert to their minima and 
begin to rise again as p continues to rise. 

The curve in Fig. 6 shows that the expected cost (the 
left axis), in units of the cost of a false alarm, for each 
attempt to use the system when the two IDSs are oper- 
ating rises as p rises. 

Consider again the environment of the 1998 DARPA 
Off-line Detection Evaluation to estimate the value of 
p = 6.52 X 10"^. Figure 6 shows that, at this prior prob- 
ability of intrusion, the best decision is to respond to an 
alarm from either IDS, the expected cost is less than 
0.003, which is less than a third the expected cost on a 
single IDS, and the best setting for each IDS is at 
a= 15x10"^ (with l-i3 = 0.72). 

The results from the analysis with two IDSs can be 
compared with the results for a single IDS as shown in 
Fig. 7. As can be easily seen, two IDSs are better than 
one over the whole range that two are better than none. 
The maximum difference in the value of two over one 
occurs at the point where the single IDS is no better 
than no IDS, i.e., dAp = 0.007. 

This result shows the limitations of the "convex hull" 
approach to evaluating multiple IDSs. Provost and 
Fawcett [15] recommend evaluating multiple IDSs by 
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Fig. 7. Expected value of two identical IDSs over one and none for 
different values of;? (at C= 500). 

finding the convex hull of their ROC curves. They then 
argue that this convex hull represents the performance 
that could be gained from using both IDSs. If any part 
of an IDS's ROC curve is on the convex hull of all 
ROC curves, then that IDS is the best one to use for 
some combination oip and C, the prior probability of 
intrusion and the cost ratio. However, their method fails 
to account for the synergistic effect that multiple IDSs 



offer to provide a more effective ROC curve than any 
single curve. Furthermore, only crossing ROC curves 
will produce different parts of the convex hull from dif- 
ferent IDSs. Identical IDSs do not cross, so the convex 
hull is the same as the single IDS. Yet two IDSs are 
clearly better than one. The following section illustrates 
this more dramatically, when a dominated IDS is added 
and the two are better than either one individually. 

3.2 Two Different IDSs 

A similar analysis could be conducted for two differ- 
ent, independent IDSs. This is the more likely case, 
since it is more likely to find two independent IDSs 
with different ROC curves than with identical ROC 
curves. Suppose, for instance, that both IDS "C" and 
IDS "£" from Fig. 1 were available for use and that 
each provided an independent assessment of whether 
an attempt to use the system was an intrusion or not. We 
saw in Sec. 2 that IDS C's performance dominated that 
of IDS "£". However an analysis of the double IDS 
with both shows that both can be used to provide a 
lower expected cost than either 

The optimal operating points and expected cost of 
the double IDS with both IDS "C" and IDS "£" are 
shown in Fig. 8. With C= 500, the combination of the 
two different IDSs is better than none over a range on 
the prior probability of intrusion, /?, from 3.8 X 10"'' to 
0.015. If/* is below the lower limit, it is best to never 
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respond to an alarm. Ifp is higher than the upper limit, 
it is best to respond to every attempt to use the system 
as if it were an intrusion. In between these limits, the 
IDS with two different IDSs behaves slightly different- 
ly from the one with identical IDSs. For values ofp up 
to 1.75 X 10"' it is best to respond only if both IDSs 
give an alarm; as p increases above this value up to 
3.7 X 10"^ it is best to respond only if IDS "C" (the bet- 
ter IDS) gives an alarm; above this value ofp, it is best 
to respond to an alarm from either IDS. 

As the prior probability of intrusion increases, the 
optimal settings of the false alarm probabilities of the 
two IDSs change as shown by the right-hand axis in 
Fig. 8. As the value of p increases above 1.75 x 10"', 
the optimal false alarm rate of the better IDS (IDS "C") 
reverts to its minimum. As the value ofp continues to 
increase above 3.7 x 10"^, the optimal false alarm rates 
for both IDSs drop and begin to rise again as p contin- 
ues to rise. 

Consider again the environment of the 1998 DARPA 
Off-line Detection Evaluation to estimate the value of 
p = 6.52 X 10"^. Figure 8 shows that, at this prior prob- 
ability of intrusion, the best decision is to respond to an 
alarm from either IDS, and the expected cost is about 
0.005, which is a little over half the expected cost with 
a single IDS like IDS "C". At this prior probability of 
intrusion, the best setting for IDS "C" is at a = 1 5 X 
10"^ (with 1 - i3= 0.72) and for IDS "£" is at a= 6 x 
10^ (with 1-^8=0.50). 

The results from the analysis with two different IDSs 
can be compared with the results for the single better 
IDS (IDS "C") as shown in Fig. 9. As can be easily 
seen, even though IDS "£" is dominated by IDS "C", 
the two different IDSs used together are better than IDS 
"C" over the whole range that the two are better than 
none (except for the range where the optimal decision 
is to respond only to alarms from IDS "C," in which 
case the double IDS has the same expected cost as IDS 
"C"). The maximum difference in the value of two dif- 
ferent IDSs over IDS "C" occurs at the point where IDS 
"C" is no better than no IDS, i.e., at j? = 0.007. Figure 9 
also shows that the double IDS made up of two IDSs 
identical to IDS "C" (the better part of the duo) is bet- 
ter than the double IDS made up of the two different 
IDSs "C" and "£". 

3.3 Two IDSs, One With No False Alarms 

Wagner and Dean [16] describe an approach to intru- 
sion detection using static analysis. They claim three 
advantages: a high degree of automation, protection 
against a broad class of attacks, and elimination of false 
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Fig. 9. Analysis of two different IDSs (at C= 500). 

alarms. The most important class of attacks for which 
their approach is applicable is buffer overflows, which 
accounted for at least half of 1999 CERT advisories 
[17]. They also claim that their approach is able to 
detect Trojan horses in trusted software, any dynamic- 
linking attack, and format string attacks. They recom- 
mend that their approach "should not be used as the 
sole defense against any of these attacks, but instead 
should be used to complement other techniques." This 
makes it an ideal case to study in conjunction with one 
of the IDSs describe in this paper 

Wagner and Dean [16] do not give any detection 
probability information in their paper In a subsequent 
e-mail to one of the authors, Wagner stated that no esti- 
mate of detection probability is available. So, for pur- 
poses of this analysis, assume that an IDS based on stat- 
ic analysis could detect 60 % of the attack attempts 
with a conditional probability of detection of 0.50. 
Thus, when considering all attack possibilities, an IDS 
based on this approach would have an operating point 
of a = 0.00 at 1 - i3 = (0.60)(0.50) = 0.30. 

Consider the operation of an intrusion detection sys- 
tem that consists of a Wagner and Dean [ 1 6] "zero false 
alarm" IDS combined with IDS "C" from Fig. 1. The 
optimal operating points and expected cost from opti- 
mal operation of such a system are shown in Fig. 10. 
With C = 500, the combination of IDS "C" with the 
"zero false alarm" IDS is better than no IDS over a 
range on the prior probability of intrusion, p, from 0.00 
to 0.010. Ifp is higher than the upper limit, it is best to 
respond to every attempt to use the system as if it were 
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an intrusion. However, because one of the component 
IDSs offers a positive detection probability at a zero 
false alarm probability, it is always better to respond to 
an alarm from the "zero false alarm" part of the system 
than to ignore it. For values ofp up to 9.5 x 10"*, it is 
best to respond only if the "zero false alarm" IDS gives 
an alarm. For values ofp between 9.5 x 10"* and 0.010, 
it is best to respond if either IDS gives an alarm. For 
values ofp above 0.010, it is best to respond to every 
attempt to use the system as though it were an intrusion. 

The "zero false alarm" IDS should always be operat- 
ed at its ROC point (0.00, 0.30). As the prior probabil- 
ity of intrusion increases, the optimal settings of the 
false alarm probabilities of the IDS "C" portion of the 
IDS changes as shown by the right-hand axis in Fig. 10. 

Consider again the environment of the 1998 DARPA 
Off-line Detection Evaluation to estimate the value of 
p = 6.52 X 10"^. Figure 10 shows that, at this prior prob- 
ability of intrusion, the best decision is to respond to an 
alarm from either IDS, and the expected cost is about 
0.0065, which is about two-thirds the expected cost 
with a single IDS "C". At this prior probability of intru- 
sion, the best setting for the IDS "C" portion of the sys- 
tem is at a = 15 x 10"^ (with 1 - ^ = 0.72). 

The results from the analyses of all three dual IDSs 
are shown in Fig. 1 1 . For very low prior probabilities of 
intrusion, below about 2 x 10"^ for the dual IDS with 
IDSs "C" and "£" and 1 x 10"" for dual identical IDS 
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"C's, the dual IDS with a "zero false alarm" (0 FA) IDS 
is better than one or both of the other dual IDSs ana- 
lyzed in this paper. Both of the others are better (as long 
as they are better than no IDS) for probabilities above 
2 X 10"^. The maximum difference in expected value (if 
C = 500) is atp = 0.01, the point at which the dual IDS 
with a "zero false alarm" component is no better than 
no IDS. 
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4. Composite ROC Curve For Multiple 
IDSs 

Results from the analyses in Sec. 3 indicate that it 
might be possible to represent the perft)rmance of two 
independent IDSs in a single, composite ROC curve. 
That is, since the expected cost and ranges on optimal 
operating points from the analysis of two independent 
IDSs can be represented in the same manner as those 
results for a single IDS, it might be possible to summa- 
rize the performance of the two IDSs in one ROC 
curve. Section 9 shows that it is indeed possible to 
derive a single, composite ROC curve from the ROC 
curves of two independent IDSs. 

The composite ROC curve displays the performance 
of the combination of two component IDSs in two 
parameters, a and ^8. This composite ROC curve is 
interpreted in the same way as any ROC curve. The val- 
ues of a and ^8 are functions of the parameters of the 
component IDSs, «!, ^y, a^, and ^82- The functional 
form, however, depends on the optimal decision rule 
used to respond to alarms from the component IDSs. 
Recall that the decision rule could be any one of the fol- 
lowing: 1) respond only if both component IDSs indi- 
cate an alarm, 2) respond only if one particular compo- 
nent IDS indicates an alarm, 3) respond only if the 
other component IDS indicates an alarm, or 4) respond 
if either IDS indicates an alarm. 



In cases where the optimal decision rule is to respond 
to a single IDS's alarm, the values of a and fi are equal 
to those of the single IDS whose advice is followed. 
When the optimal decision rule is to respond only when 
both component IDSs indicate an alarm, the parameters 
of the composite ROC curve are: a= aia2, and 
j8 = j8i + ^82 - f^ijii- When the optimal decision rule is 
to respond to an alarm from either component IDS, the 
effective parameters of the composite ROC curve are: 
a= ai + 02- OiiOi, and P = ^81^82. In any case, the 
expected cost (EC), in terms of the prior probability of 
intrusion, p, and the cost ratio, C, from operating at a 
point on the ROC curve is: EC = (1 -p)a + Cpfi. 

Applying these results to the analysis with two dif- 
ferent IDSs from Sec. 3.2 gives the composite ROC 
curve with the points shown in Table 1. This ROC 
curve is displayed graphically in Fig. 12. 

Notice that four points from the ROC of IDS "C," 
which are shown bounding the solid lines in Fig. 12, are 
points on the composite ROC. The fifth point (15 X 
10"^, 0.72) is not on the composite ROC because it is 
not on the convex hull of all points. Section 8.2 
describes how the convex hull of points is determined 
from a set of points. All possible points for the compos- 
ite ROC are generated by considering all four decision 
rules. 

Since the performance of a compound IDS com- 
posed of two independent IDSs can be represented as a 



Table 1. Composite ROC curve for two different IDSs 



IDS 1 (IDS 


"O 


IDS 2 (IDS 


"£") 




Composite 




«! 


1-A 


"2 


l-fe 


Respond to: 


a 


1-/3 


1.00 


1.00 


1.00 


1.00 


Always 


1.00 


1.00 


15x10"^ 


0.72 


10x10^ 


0.52 


Either 


115.0x10"^ 


0.8656 


15x10"^ 


0.72 


6x10^ 


0.50 


Either 


75.0x10"^ 


0.8600 


15x10"^ 


0.72 


4x10^ 


0.45 


Either 


55.0 X 10"^ 


0.8460 


15x10"^ 


0.72 


2x10^ 


0.35 


Either 


35.0x10"^ 


0.8180 


10x10"^ 


0.70 


2x10^ 


0.35 


Either 


30.0x10"^ 


0.8050 


7x10"^ 


0.68 


2x10^ 


0.35 


Either 


27.0x10"^ 


0.7920 


10x10"^ 


0.70 






IDS 1 


10.0x10"^ 


0.7000 


7x10"^ 


0.68 






IDS 1 


7.0 X 10"^ 


0.6800 


5x10"^ 


0.66 






IDS 1 


5.0 X 10"^ 


0.6600 


2x 10"^ 


0.60 






IDS 1 


2.0 X 10"^ 


0.6000 


15x10"^ 


0.72 


10x10^ 


0.52 


Both 


15.0x10"* 


0.3744 


10x10"^ 


0.70 


10x10^ 


0.52 


Both 


10.0x10"* 


0.3640 


7x10"^ 


0.68 


10x10^ 


0.52 


Both 


7.0 X 10"* 


0.3536 


10x10"^ 


0.70 


6x10^ 


0.50 


Both 


6.0 X 10"* 


0.3500 


7x10"^ 


0.68 


6x10^ 


0.50 


Both 


4.2 X 10"* 


0.3400 


5x10"^ 


0.66 


6x10"* 


0.50 


Both 


3.0x10"* 


0.3300 


2x 10"^ 


0.60 


6x10"* 


0.50 


Both 


1.2x10"* 


0.3000 


2x10"^ 


0.60 


4x10^ 


0.45 


Both 


0.8x10"* 


0.2700 


2x10"^ 


0.60 


2x10^ 


0.35 


Both 


0.4 X 10"* 


0.2100 


0.00 


0.00 


0.00 


0.00 


Never 


0.0 


0.0000 



463 



Volume 108, Number 6, November-December 2003 

Journal of Research of the National Institute of Standards and Technology 



0.90 



"S 0.80 



£ 0.70 
o 

o 

% 0.60 

Q 

**— 

o 

2r 0.50 



J2 
CD 

O 



i 

0.40 



;; 

0.30 - 



0.20 



H 1 



0.0005 0.001 

False alarm probability, alpha 

Fig. 12. Composite ROC curve for an IDS consisting of IDS "C and IDS "£". 



single composite ROC curve, its analysis can be per- 
formed in exactly the same way as the analysis of a sin- 
gle IDS. That is, an analysis of the ROC curve in Fig. 
12 produces the same results as displayed in Fig. 8 (in 
Sec. 3.2). Furthermore, the composite ROC curve for 
two IDSs can be combined with the ROC curve from 
another independent IDS to produce a composite ROC 
curve for the combined IDSs. That is, the same method 
used to combine two independent ROC curves can be 
applied iterative ly to combine any number of independ- 
ent ROC curves. 

This finding suggests the following, six-step proce- 
dure to evaluate any number of independent IDSs: 

Step 1 . Determine the equivalent convex ROC curve 
for each IDS. (See Sec. 8. 1 for the method to determine 
the convex equivalent for concave sections of an ROC 
curve.) 

Step 2. For a pair of IDSs, specify all combinations 
of points on the two ROC curves. 

Step 3. Determine the values of parameters a and j3 
for all combinations of points from Step 2 under each 
of these two decision rules: 1) respond only if both 
IDSs indicate an alarm and 2) respond if either IDS 
indicates an alarm. 



Step 4. Identify the convex hull of the set of all 
points. The set consists of the points (a, I - p) on each 
component IDS's ROC curve, all of the points deter- 
mined in Step 3, and the endpoints, (0, 0) and (1, 1). 
The convex hull of these points is the composite ROC 
curve for the two independent IDSs. 

Step 5. Repeat Steps 2 through 4 until all component 
IDSs are included in the composite ROC curve. This 
might be done by adding each component's ROC 
curve, in turn, to the composite ROC curve or by first 
combining pairs of ROC curves and then combining the 
resulting composite curves. 

Step 6. Use the final composite ROC curve to ana- 
lyze the combined performance of all of the independ- 
ent IDSs. Notice that one must keep track of the com- 
bination rules used to generate each point on the com- 
posite ROC curve in order to determine the settings of 
the component IDSs that produce each point. 



5. Conclusions, Recommendations, and 
Extensions 

The analysis in this paper demonstrates that the most 
commonly recommended methods for evaluating and 
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comparing IDSs are flawed. IDSs should not be evalu- 
ated based on the areas under their ROC curves, their 
distances from a goal, or their false alarm rates. 
Evaluations should be based on expected costs that 
reflect: the cost of a false alarm, the cost of a failure to 
detect (or a ratio of these costs), and the prior probabil- 
ity of intrusion. Furthermore, the operating point of an 
IDS, its probability of a false alarm (a) and probability 
of missed detection (fi), should be established to mini- 
mize the expected cost. 

When considering the operation of multiple IDSs, 
the ROC convex hull [15] is an insufficient guide for 
determining how to make best use of multiple IDSs. 
The ROC convex hull can be used to determine the 
combinations of costs and prior probabilities of intru- 
sion for which one IDS is preferred over another It 
does not indicate the degree of preference. However, 
unless one of the IDSs is worthless, it is better to use 
the IDSs in combination than to use a single IDS. The 
performance of the combination of IDSs can be repre- 
sented in a composite ROC curve that can be used for 
analyses. 

The methods described in this paper are suitable for 
development into a decision support tool that could be 
used by a system administrator to choose among IDSs, 
to indicate the best use of a single IDS or any combina- 
tion of independent IDSs, and to set the operating 
parameters of an IDS for optimal performance in a 
given environment characterized by costs and the prior 
probability of intrusion. This tool could also be used by 
IDS developers to evaluate design tradeoffs that lead to 
different performance. 



Different types of intrusions can be analyzed by 
extending the decision tree in Fig. 2 to include event 
nodes that show explicitly the type of intrusion with a 
different ROC curve applied to each. For example, 
ROC curves for the Columbia IDS were determined for 
the four types of attacks included in the 1998 DARPA 
off-line intrusion detection evaluation [3,4,10]. 
Columbia's ROC curve is worst against denial of serv- 
ice attacks, a little better against remote-to-local and 
user-to-root attacks (the same ROC curve applies for 
both of these types of attacks), and substantially better 
against surveillance and probing attacks. 

The analysis in this paper handles an evolving attack 
of intrusion and response only implicitly. The probabil- 
ity of detection is interpreted as the probability that an 
intrusion is detected before it does any damage, and the 
response is interpreted to be both immediate and effec- 
tive. Any intrusion that is not detected and countered 
immediately and effectively is modeled in the single 
path with no response to an intrusion. That is, Cp is the 
expected cost considering all levels of damage that may 
occur before the intrusion is neutralized. A more 
detailed analysis could show a time sequence of events 
that corresponded to delayed detections, delayed 
responses, their probabilities, and their effectiveness. 

Although the analysis in this paper concentrates on 
the case where costs are associated only with errors, the 
method extends to analyses with other costs, as out- 
lined in Sec. 8. 



6. Appendix A: Analysis For a Compound IDS Witli a Single Decision 

This appendix shows the analysis for a compound IDS composed of two independent IDSs when the response 
decision is made only once on the basis of both reports. For two independent detectors: 

PiAl, A2|I) = /'(A1|I) /'(A2|I) = (1 - PiXl - P^ypi ; PiAi, A2|NI) = /'(A1|NI) /'(A2|NI) = UiU^ ; 
PiAl, NA2|I) = /'(A1|I) /'(NA2|I) = (1 - Pi) P^ ; PiAl, NA2|NI) = /'(A1|NI) /'(NA2|NI) = ai(l - oQ ; 
P(NAl, A2|I) = /'(NA1|I) PiA2\l) = ^(1 - Pi) ; /'(NAl, A2|NI) = /'(NA1|NI) /'(A2|NI) = (1 - ai)«2 ; 
/•(NAl, NA2|I) = /'(NA1|I) /'(NA2|I) = PiP^ ; /'(NAl, NA2|NI) = /'(NA1|NI) /'(NA2|NI) = (1 - a^ (1 - oQ. 

So, by Bayes' Theorem (see [13]), the values of the parameters are: 

qi = P{Al, A2\l)P{l)/pi =p{l - A)(l - p2)/Pi ; 1 - ^1 = ^Al, A2|NI)/'(NI)//?, = (1 -p)aia,lpi ; 
q^ = PiAl, 'NA2\l)P{T)/p2=pil - A) PJPi ; 1 - ^2 = ^(Al, NA2|NI)/'(NI)//'2 = (1 -/?)ai(l - oQ/p2 ; 
q, = /-(NAl, A2\l)Pil)/p, =M(1 - p2yP3 ; 1 - ^3 = ^(NAl, A2|NI)/'(NI)/;?3 = (1 -p) (1 - ai)a2/p, ; 
q, = P(NAl, NA2\l)P{l)/p,=ppipJp, ■,l-q, = /'(NAl, NA2|NI)/'(NI)/a = (1 -p) (1 - aj (1 - oQ. 
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Expected Cost =/»i Min{\-qi, Cqi} + P2 Minjl-gj, C^j} + P3 Minjl-^j, Cq^} + P4 Min{l-g4, Cq^} 

=p, Min{{\-p)a,a2lp, , Cp(l-pi)(l-p2)/pi} +P2 Min{(l-/?)a,(l-a2)//?2 , Q?(l-A) PJPi} 
+ p, Mm{(\-p)(l-ada2/p, , CMCl-AVft} + A Mm{(\-p)(\-ad(l-oQ/p„ Cpp^pM} 

= Mm{il-p)a,a2 , Cpil-pdil-pi)} + Min{(l-;?)a,(l-a2) , Q?(l-A) A} 
+ Mm{(l-p)(l-ada2, Cpm-Pi)} + Min{(l-;j)(l-a,)(l-«2), CMAI- 



7. Appendix B: Analysis For a Compound IDS With Sequential Decisions 

This appendix shows the analysis for a compound IDS composed of two independent IDSs when the response 
decision is made sequentially after each component IDS's report. The decision tree for two IDSs with sequential deci- 
sions, one after each IDS's report, is shown in Fig. 13. 
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Fig. 13. Two detectors, sequential decisions. 
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The analysis is as follows. For two independent detectors: 

p, = P{M\l)P{l) + P(A1 |NI)P(NI) = (1 - PdP + «i(l -P); ^-Pi=P<P + ii- ai)(l -P); 

p, = P(A2\l)P(l) + P(A2\m)P(Nl) = (1 - P2)P + 0C2(l -p); 1 -p^ = P^ + (1- a,){\ -p); 

p, = P(l\A\) = P(A1 |I)P(I)/ P(A,) = (1 - P,)p/p,; \-p,= a,(l -p)/p,; 

p, = P(I|NA1) = P(NA1|I)P(I)/ P(NA1) = Pj,/(1 -p,); 1 -p, = (1- a,)(\ -p)/(l -p,). 

q, = P{l\A\, A2) = P(A1, A2|I)Pa)//'(Al, A2) = p{\ - A)(l - ^^ypj^^ ; 1 - ^i = (1 -p)a,a,lp^i, 
q, = P(l\A\, NA2) = P(A1, NA2|I)P(I)/P(A1, NA2) =p(l - A) A/Pi(l -Pi) ; 

1 - ^2 = (1 -p)aiil - a^ypiil -P2) ; 
q, = P(I|NA1, A2) =M(1 - A)/(l -Pi)P2 ; 1 - ^3 = (1 -P)(^ - «i)«2/(l -pdPi ; 
q,=pPA/0 -pdO -Pi) ; 1 - ^4 = (1 -P)(i - ai)(l - oQ /(I -;j,)(l -^'i). 

Expected Cost =pi Mm[(l-p)ai/pi , Mm{{\-p)aiaj)lpip, C/'(l-A)(l-A)//'i/'2} + (1 -Pi) Min{(l-/))ai(l-a2)//'i(l- 
p,) , Cpil-p,) pjp.il -p,)}] + (1 -pd Min[(l -p)il - «,)/(! -p,) , Min{(l-;j)(l-a,)«2/(l -pd, CM(1-A)/(1 " 
p,)} +Mm{(\-p)(l-a,)(l-a2)/(l -p,), CpPAK} -Pi)}} 

= Min[(l-;?)a, , Mm{{\-p)a,a, , Qj(1-A)(1-A)} + Min{(l-;?)a,(l-a2) , Q?(l-A) A}] 

+ Min[(l-/?)(l-a,), Min{(l-;?)(l-a,)a2, Cpm-Pi)} + Min{(l-;j)(l-a,)(l-«2), CpPA}l 

Now, 

(!-;?)«, = Min[(l-/?)a, , Mm{{\-p)a,a, , Cpil-^^Xl-p,}} + Min {(!-/?)«,( l-a^) , Q?(l-A) A)] 

only if: 

(\-p)ai < (\-p)aia2 + (\-p)ai(l-a2) 

< {l-p)ai(X2 + {i-p)(Xi - {l-p)aia2 - {i-p)(Xi 

< 0, which is a contradiction. 



And, 



only if 



{l-p){l-ad = Min[(l-;j)(l-a,), Min{(l-;j)(l-a,)a2, CM(l-A)} + Min{(l-;j)(l-a,)(l-a2), CMA}] 



(\-p)(l-a,) < (l-p)(l-a,)a2 + (l-;?)(l-a,)(l-«2) 

< (l-p)(l-a,)a2 + (l-p)0-a,) - (l-p)(l-a,)a,-(l-p)(l-a,) 

< 0, which is a contradiction. 

Therefore, 

Min[(l-;?)a, , Mm{il-p)a,a, , Qj(1-A)(1-A)} + Min{(l-;?)a,(l-a2) , Q?(l-A) A}] 
+ Mm[(l-p)(\-a,), Mm{(l-p)(l-a,)a„ CpP.il-P^)} + Min{(l-;j)(l-a,)(l-«2), Q?AA}] 

= Mm{il-p)a,a, , Q?(l-A)(l-A)} + Min{(l-;?)a,(l-a2) , Q?(l-A) A) 
+ Min{(l-;?)(l-a,)a2, CM(l-A)} + Min{(l-;j)(l-a,)(l-a2), CMA) 

The last expression is identical to the expression for the expected cost for deciding on the response to two inde- 
pendent detectors after the results from both are known, as derived in Sec. 6. Since there is no incremental cost to 
getting the second detector's report, the expected cost from using an IDS composed of two independent detectors is 
the same regardless of whether the response decision is made sequentially after each report or if it is made only once 
on the basis of both reports. 
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8. Appendix C. Simplification of tlie Decision Analysis and Geometry of ROC 



8.1 Removal of Embedded Decision 

Consider the decision tree shown in Fig. 2 and its expected cost: Mm{Cpp, (1 - a)(l - p)} + Min{C(l - p)p, 
0^(1 ~P)}- The four possible combinations of expressions are the expected costs from following each of four strate- 
gies: never respond, always respond, follow the IDS's recommendation (i.e., respond only to alarms), and do the 
opposite of what the IDS recommends (i.e., respond only to no alarm). Recognize that the strategies of always 
responding and never responding reduce to the strategy of following the IDS's recommendation for an IDS operat- 
ing at the points (1,1) and (0, 0), respectively. 

Thus, the decision tree in Fig. 2 can be simplified to remove the embedded decision if it can be shown that it is 
never optimal to do the opposite of the IDS's recommendation. This is the case for a convex ROC curve (i.e., one 
where 1 - j3 > a for at least one combination of a and p). One would either follow the IDS's recommendation or one 
would always respond or never respond. Assume that it is optimal to follow the IDS's recommendation, then: 

"Follow" is better than "Never Respond": Cp>i\-p) a+ CpP; 
"Follow" is better than "Always Respond": l-p>{l -p) a + Cpfi; and 
"Follow" is better than "Opposite": Cp(l - p) + (I -p)(l -d)>(l-p)a+ Cpp. 

Now, 

i. Cp > (1 -p) a + CpP^ Cp(l - i3) > [(1 -p) a + Cpp](l - p); and 
ii. 1 -i? > (1 -p) a + CpP =^ (1 -p)il -a)> [(1 -p) a + Cpp\il - a). 

Thus, 

[(1 -p) a+Cpp](l-l3) + [(1 -p) a+Cpp](l -«)>(! -p) a+CpP 
=^ Cp(l -p) + (l -p)(l -«)>(! -p) a+CpP. 

Now, 

[(1 -p) a+CppJ(l-p) + [(1 -p)a + Cpp\(l - a) > (1 -p) a+Cpp 
if (1 - i3) + (1 - a) > 1, that is for 1 - i3> a. 

However, if an IDS has a concave ROC curve (i.e., one where I- p< a), then cost could be minimized by doing 
the opposite of what the IDS recommends. It is important to notice that a concave ROC can be as valuable as a con- 
vex one. Any IDS with a ROC curve that differs from the ROC curve of no IDS, I - P= a, offers some information 
and is better than no IDS for some combination of cost and prior probability of intrusion. Doing the opposite of what 
the IDS recommends is optimal if: 

"Opposite" is better than "Never Respond": (1 -p)il - a) + Cpil -p}<Cp; 
"Opposite" is better than "Always Respond": (1 -/')(1 - a) + Q?(l - jS) < 1 -/?; and 
"Opposite" is better than "Follow": (1 -p){\ - a) + Cp(\ -p)<(l-p)a+ Cpfi. 

Now, 

iii. (1 -p){l -a) + Cp{\ -p)<Cp^[{\ -p){\ -a) + Cp{l - p)\P < CpP; and 
iv. (1 -p)(l -a) + Cp(l-I3)< 1 -p =^ [(1 -p)(l -a) + Cp(l-p)]a<(l -p)a. 

Thus, 



468 



Volume 108, Number 6, November-December 2003 

Journal of Research of the National Institute of Standards and Technology 



(1 -p){\ -a) + Cp{\-p)< [(1 -p){\ - a) + Cp{\ -i5)]a+ [(1 -p){\ -a) + Cp{\ - mP 
=> (1 -p){\ -a) + Cp{\ -p)<{\-p)a+ Cpp. 

Now, 

(1 -p){\ -a) + Cp{l -p)<[{l -p){l -a) + Cp{\ - i3)]a+ [(1 -p){l -a) + Cp{l - p)]l5 
if 1 < a + A that is for 1 - ^ < a. 

Furthermore, a concave ROC curve with parameters a and P is equivalent to a convex ROC curve with parame- 
ters a' and P', where a' = 1 - a and P' = I - p. The expected cost from following the convex ROC with parame- 
ters a and P is equal to the expected cost from acting opposite to the concave ROC with parameters a' and P'. 

(1 -p)a+CpP = {\ -p){\ - a) + Cp{\ -P') if 
(\-p)a+CpP = {\-p){\-{\-d}\ + Cp{\-{\-P)l 
(1 -p)a + 03^8 = (1 -p)a + CpP. 

So, if the extreme points of (0, 0) and (1, 1) are considered part of the ROC curve and if points where 1 - ^8 < a are 
transformed by setting a ' = 1 - a and P' = \ - P, then the embedded decision can be eliminated from the decision 
tree, and the expected cost is simply: a(l -p) + CPp. The structure of the simplified decision tree is shown in Fig. 
16. 

Summarizing, an IDS that includes the point (0, 1) is a perfect IDS. An IDS that includes the point (1, 0) is also a 
perfect IDS, since, by doing the opposite of its recommendation, one achieves performance equivalent to the IDS 
with the point (0, 1). An IDS that includes only the points (0, 0), (1, 1), or any along the straight line connecting these 
two, is a worthless non-IDS. All other IDSs are better than none but not as good as perfect. 

8.2 ROC Convex Hull 

Provost and Fawcett [15] describe a method that they call the ROC convex hull method for evaluating IDSs. 
Simply stated, the method builds a composite ROC curve from several ROC curves by taking the convex hull of all 
available curves. This identifies the ROC curves that are optimal for some cost and prior probability combinations. 
They do not recognize the value of concave ROC curves, but their method is trivially extended to include the con- 
vex equivalents of concave ROCs. 

In the case of IDSs that offer sets of points for an ROC curve, the convex hull will be piecewise linear, and the 
determination of the optimal setting is the linear programming problem with the feasible region bounded by line seg- 
ments connecting the points on the convex hull of ROC curves and the extreme points of (0, 0) and (1, 1), and the 
objective fiinction is to minimize: a(l -p) + CPp. 

The geometry of the situation is shown for a ROC curve with a single point in Fig. 14. The IDS is better than no 
IDS as long as the slope of the cost minimization objective fiinction is between the slope of the line through (0, 0) 
and (Uq, 1 - Pq) and the slope of the line through (Oq, 1 - Pq) and (1, 1). This suggests several metrics for the ROC 
curve. A ROC curve is preferred over no curve over a greater range of objective fiinctions if 9 is smaller (O.Sit < 9< 
k). Alternatively, a ROC curve is preferred over no curve over a greater range of objective fiinctions if sin(0) is larg- 
er (0 < sin(0) < 1). This is consistent with the statement that a ROC curve is preferred over no curve over a greater 
range of objective fiinctions if the area. A, under the ROC curve is larger (0 < ^ < 1). Since the area under the "no 
IDS" line is equal to 0.5, the metric ^' = ^ - 0.5 is probably a better metric than A. The angle 9 can be determined 
from the coordinates of the ROC point as follows: 

tan 9 = (mj - 'WiVCl + ^z 'Wi), where Wi = (1 - Po)/oCq and Wj = Po/{\ - oCq). Solving for 9, 

9= tan"' {[Oq Po-i^ - c^o) (1 - PoWlc^oi^ - c«o) + Poi^ - Po)]}- The area between the IDS's ROC curve and the curve 

for no IDS is: ^' = 0.5 (l-p^- «o). 
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Fig. 14. Convex hull of a ROC curve with a single point. 

However, if anything is known about p or C, the ranges on these parameters over which the IDS is better than no 
IDS are probably better metrics than any of the four mentioned above. Operating at the point on the ROC curve is 
best if the following two conditions hold: 

i. Cp > (1 -p)cc,) + CppQ , and 
ii. l-p>il-p)ao + Cpl5o. 

Solving for C: 

[(1 -pyp)ia,/il - M < C < [(1 -p)/pm - Oo)/ p,]. 
Or, solving for p: 

aJ[C(l -l3o) + a,]<p<(l- ao)/(l -cCo + Cpo). 

A similar result obtains for the case where the ROC curve is more than one point, which is illustrated in Fig. 15. 
Here, the IDS is better than none if: 

[(1 -pyp)(aj(l - pd] < C < [(1 -pypm - a2yp2)], or, equivalents, if 
aJ[Cil - A) + oci] <p<il- a2)/(l - «2 + Cpi). 

Here, the angle over which the IDS is better than no IDS is: 

= tan-' {[a, jS^ - (1 - c^) (1 - A)]/[«i(l - «2) + ^(1 - A)]}- 

8.3 Costs On All Outcomes 

A similar analysis can be conducted for the general case where costs are associated with all four combinations of 
intrusion and response. For an analysis based on expected cost, the actual costs can be transformed by first subtract- 
ing the cost of one outcome from the costs of the others and then dividing all costs by one of the remaining non-zero 
costs. Now the costs of the outcomes are all expressed as a linear function of the original costs. Since the expecta- 
tion of a linear function of a random variable is the same linear function of its expectation, these operations preserve 
the expected cost calculation. 
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Fig. 15. Convex hull of a ROC curve with multiple points. 



This transformation results in the event tree for the case of general costs shown in Fig. 16. Costs of event combi- 
nations are scaled relative costs. They are relative to the cost of no response and no intrusion (the quiescent cost). 
They are scaled to units of the amount that the false alarm cost exceeds the quiescent cost. With this cost structure, 
the objective function is to minimize: K{1 - p)p + a(l -p) + CPp. There are now two cost ratios, C and K. Solving 
for the range over which the IDS is better than no IDS in terms of the prior probability,/), yields: 

«!/[«! + (C - K)(l - Pi)] <p<(l- oQ/[l - 02 + (C - K)I32]. 

Let D = C-K, the difference in cost between failing to respond to an intrusion and responding to an intrusion. The 
usual condition is that C> Kor, alternatively D> (i.e., it is better to respond to an intrusion than to not respond). 
Solving for D, the range over which the IDS is better than no IDS in terms of this cost difference is: 

«i(l -P)m - Pdp] <D<(1- a2)(l -pyp^. 
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Fig. 16. Event tree with general costs. 



9. Appendix D. Composite ROC Curves 

This appendix shows that a composite ROC curve can be obtained from two component ROC curves. The com- 
posite curve depends on the component ROC curves and the decision rule used to respond to alarms from either or 
both of the component IDSs. This analysis assumes that each component ROC curve is convex or the convex equiv- 
alent of a concave ROC curve (as described in Sec. 8). In this case, the decision analysis can be simplified to con- 
sider only the cases of responding to alarms. In general, one must consider the decision rules of 1) responding only 
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if both component IDSs indicate an alarm, 2) responding only if one of the components IDSs (labeled IDS 1) indi- 
cates an alarm, 3) responding only if the other component IDS (labeled IDS 2) indicates an alarm, or 4) responding 
if either IDS indicates an alarm. 

In terms of the parameters of the IDSs, the prior probability of intrusion, p, and the cost ratio, C, described in Sec. 
2, the expected costs from following the decision rules are as follows. The expected cost of responding to both alarms 
is: 

£Cb = (1 -p)a,a, + Cp{\- Pd A + Cpm- Pi) + CpPA 
= (l-;j)a,a2 + Qj(A + A-AA)- 

The expected cost of responding to IDS 1 's alarms is: 

EC, = (1 -p)a,a2 + (l-/?)a,(l-a2) + Cp^- A) + CMA 
= (1 -/?)«! + CpP,. 

The expected cost of responding to IDS 2's alarms is: 

EC2 = (1 -p)a,a2 + Q?(l-A) A + (1 -^'Xl - «i)0!2 + CMA 
= i\-p)a, + CpP2- 



The expected cost of responding to either alarm is: 

ECj, = (1 -p)aia2 + (1 -i9)«i(l -oQ + il 
= (1 -/»)(«! + 02 - aiOj) + CpPipj' 



- p){\ - ad(h + Cp^^P^ 



Notice that the cases where the decision rule is to respond to a single IDS's alarm are equivalent to using the sin- 
gle IDS. Thus, the parts of the composite IDS's ROC curve for the conditions under which the optimal decision rule 
is to respond to just a single IDS's alarm is the same as that IDS's ROC. For conditions where the other decision rules 
are optimal, the composite ROC will have effective values of a and fi that can be calculated from the parameters of 
the component IDSs. Compare the expected value results above with the expected value equations for a single IDS 
as given in Sec. 8.1. Notice that the factor associated with (1 -p) is the effective value of a, and the factor associat- 
ed with Cp is the effective fi. Therefore, when the optimal decision rule is to respond only when both component 
IDSs indicate an alarm, the effective parameters of the composite ROC curve are: a^ = ayOq^ and i^B = A + A ~ P3i- 
When the optimal decision rule is to respond to an alarm from either component IDS, the effective parameters of the 
composite ROC curve are: a^= ay + Oq^- ayOq^ and ^ = PiPj- 
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